703 research outputs found
A PDTB-Styled End-to-End Discourse Parser
We have developed a full discourse parser in the Penn Discourse Treebank
(PDTB) style. Our trained parser first identifies all discourse and
non-discourse relations, locates and labels their arguments, and then
classifies their relation types. When appropriate, the attribution spans to
these relations are also determined. We present a comprehensive evaluation from
both component-wise and error-cascading perspectives.Comment: 15 pages, 5 figures, 7 table
Discourse parsing: Inferring discourse structure, modeling coherence, and its applications
Ph.DDOCTOR OF PHILOSOPH
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
In this paper, we focus on the problem of Medical Visual Question Answering
(MedVQA), which is crucial in efficiently interpreting medical images with
vital clinic-relevant information. Firstly, we reframe the problem of MedVQA as
a generation task that naturally follows the human-machine interaction, we
propose a generative-based model for medical visual understanding by aligning
visual information from a pre-trained vision encoder with a large language
model. Secondly, we establish a scalable pipeline to construct a large-scale
medical visual question-answering dataset, named PMC-VQA, which contains 227k
VQA pairs of 149k images that cover various modalities or diseases. Thirdly, we
pre-train our proposed model on PMC-VQA and then fine-tune it on multiple
public benchmarks, e.g., VQA-RAD and SLAKE, outperforming existing work by a
large margin. Additionally, we propose a test set that has undergone manual
verification, which is significantly more challenging, even the best models
struggle to solve
Hierarchical Side-Tuning for Vision Transformers
Fine-tuning pre-trained Vision Transformers (ViT) has consistently
demonstrated promising performance in the realm of visual recognition. However,
adapting large pre-trained models to various tasks poses a significant
challenge. This challenge arises from the need for each model to undergo an
independent and comprehensive fine-tuning process, leading to substantial
computational and memory demands. While recent advancements in
Parameter-efficient Transfer Learning (PETL) have demonstrated their ability to
achieve superior performance compared to full fine-tuning with a smaller subset
of parameter updates, they tend to overlook dense prediction tasks such as
object detection and segmentation. In this paper, we introduce Hierarchical
Side-Tuning (HST), a novel PETL approach that enables ViT transfer to various
downstream tasks effectively. Diverging from existing methods that exclusively
fine-tune parameters within input spaces or certain modules connected to the
backbone, we tune a lightweight and hierarchical side network (HSN) that
leverages intermediate activations extracted from the backbone and generates
multi-scale features to make predictions. To validate HST, we conducted
extensive experiments encompassing diverse visual tasks, including
classification, object detection, instance segmentation, and semantic
segmentation. Notably, our method achieves state-of-the-art average Top-1
accuracy of 76.0% on VTAB-1k, all while fine-tuning a mere 0.78M parameters.
When applied to object detection tasks on COCO testdev benchmark, HST even
surpasses full fine-tuning and obtains better performance with 49.7 box AP and
43.2 mask AP using Cascade Mask R-CNN
- …